Comparing System Evaluation with User Experiments for Japanese Web Navigational Retrieval

We conducted a search experiment targeting 31 users to investigate whether the performance evaluation metrics of IR systems used in test collections, such as TREC and NTCIR, are comparable to the user performance and subjective evaluation. We selected three systems with high, medium, and low performance values in terms of nDCG, MRR and Prec@10 metrics from among the retrieval systems that participated in the NTCIR-5 WEB task, and then selected three topics. The results of the experiment showed no significant differences between these systems and topics in the completion time for each search. Furthermore, none of the results of the users’ evaluations corresponded to the results of the batch system evaluations. These results indicate a need for new evaluation metrics that correspond to the users’ evaluations.